Differential Item Functioning Research Articles

Piloting Rasch model scoring of the National Eye Institute Visual Function Questionnaire in uveitis.

The National Eye Institute Visual Function Questionnaire (NEI VFQ) is a common patient-reported outcome measure (PROM) in uveitis trials. Its psychometric properties using state-of-the-art scoring based on Rasch models, a latent trait model that improves accuracy of PROMs assessment, has not yet been investigated. The study participants were recruited online from uveitis patient organizations, where individuals self-reported their uveitis diagnosis and visual acuity level. These participants then completed the NEI VFQ-25. The visual function (VF) and socioemotional (SE) subscales were psychometrically analysed in terms of item fit, targeting, internal consistency, dimensionality, and differential item functioning (DIF), using Rasch models. Criterion validity was examined based on associations between NEI VFQ person measures and recent visual acuity (VA) levels. Ninety-nine participants recruited online from uveitis patient organizations (68 women, 31 men; mean age 50 ± 15years; 46.5% self-reported receiving systematic therapy for uveitis, 0.6% NEI VFQ-25 missing data) were included. The mean difficulty of items was lower than the average person ability. None of the items demonstrated misfit to an extent that would induce noise into the measurement. The consistency metrics person reliability and person separation index of the subscales were 0.85 and 2.34 (NEI VFQ-VF), 0.86 and 2.52 (NEI VFQ-SE), respectively. There was no evidence of multidimensionality and none of the items showed DIF by gender. The differences between item and person measures were 1.44 (NEI VFQ-VF) and 1.03 (NEI VFQ-SE). NEI VFQ-25 person measures were significantly lower in participants with visual impairment (all p values ≤ 0.007). Rasch model-based scoring of the re-engineered NEI VFQ-25 demonstrates acceptable internal consistency, item fit and construct validity for assessing two key domains of quality of life in individuals self-reporting uveitis. The PROM was targeted at a higher level of difficulty than present in our heterogeneous sample.

A Framework for Detecting Both Main Effect and Interactive DIF in Multidimensional Forced-Choice Assessments

In recent decades, multidimensional forced-choice (MFC) tests have gained widespread popularity in organizational settings due to their effectiveness in reducing response biases. Detecting differential item functioning (DIF) is crucial in developing MFC tests, as it relates to test fairness and validity. However, existing methods appear insufficient for detecting DIF induced by the interaction between multiple covariates. Furthermore, for multi-category, ordered or continuous covariates, existing approaches often dichotomize them using a-priori cutoffs, commonly using the median of the covariates. This may lead to information loss and reduced power in detecting MFC DIF. To address these limitations, we propose a method to identify both main effect DIF and interactive DIF. This method can automatically search for the optimal cutoffs for ordered or continuous covariates without pre-defined cutoffs. We introduce the rationale behind the proposed method and evaluate its performance through three Monte Carlo simulation studies. Results demonstrate that the proposed method effectively identifies various DIF forms in MFC tests, thereby increasing detection power. Finally, we provide an empirical application to illustrate the practical applicability of the proposed method.

Attitude Is Everything: Examining Acceptance of Violence Items for Differential Item Functioning Across Heterosexual and Sexual Minority Youth.

Acceptance of dating violence (ADV) is a cognitive risk factor for violence perpetration and a common target of prevention programs. However, frequently used items assessing ADV are characterized by heteronormative item wording, and limited research has evaluated the degree to which ADV items function equivalently for both heterosexual and sexual minority youth (SMY). The current study sought to determine if there are differences in the way heterosexual and SMY respond to ADV survey items. Secondary data from a total of 2,014 adolescents (Mage = 16.78) were used to examine differences in ADV. Results of differential item functioning analysis indicated nonuniform differential item functioning for two of eight ADV items, with heterosexual youth being more likely to express strong levels of agreements with (a) female-perpetrated physical violence in response to male-perpetrated violence and (b) female-perpetrated violence against males broadly, relative to SMY. Although these differences were of negligible magnitude and only resulted in minimal differences in overall expected average scores, heterosexual youth were more likely to strongly accept female-perpetrated dating violence compared to SMY. Findings highlight differences in ADV item response patterns across heterosexual and sexual minority identifying youth and provide preliminary evidence for group differences in acceptance of female-perpetrated dating violence. Implications for prevention programming based on current findings include greater focus on measure adaptation and development as well as more consensus on the necessity of preventing female-perpetrated violence.

Smooth Information Criterion for Regularized Estimation of Item Response Models

Item response theory (IRT) models are frequently used to analyze multivariate categorical data from questionnaires or cognitive test data. In order to reduce the model complexity in item response models, regularized estimation is now widely applied, adding a nondifferentiable penalty function like the LASSO or the SCAD penalty to the log-likelihood function in the optimization function. In most applications, regularized estimation repeatedly estimates the IRT model on a grid of regularization parameters λ. The final model is selected for the parameter that minimizes the Akaike or Bayesian information criterion (AIC or BIC). In recent work, it has been proposed to directly minimize a smooth approximation of the AIC or the BIC for regularized estimation. This approach circumvents the repeated estimation of the IRT model. To this end, the computation time is substantially reduced. The adequacy of the new approach is demonstrated by three simulation studies focusing on regularized estimation for IRT models with differential item functioning, multidimensional IRT models with cross-loadings, and the mixed Rasch/two-parameter logistic IRT model. It was found from the simulation studies that the computationally less demanding direct optimization based on the smooth variants of AIC and BIC had comparable or improved performance compared to the ordinarily employed repeated regularized estimation based on AIC or BIC.

The Ethiopian Cognitive Assessment battery in Schizophrenia (ECAS): a validation study

Cognitive impairment is common in people with schizophrenia (PWS). To detect the presence and its consequences, cognitive measures with sound psychometric properties are needed. However, these are lacking especially in low-income countries. Hence, we developed the Ethiopian Cognitive Assessment battery in Schizophrenia (ECAS). In this study, we evaluated the psychometric properties of the ECAS in a cross-sectional study involving 350 PWS. Confirmatory factor analysis demonstrated a one-factor solution. ECAS score correlated significantly but weakly with a disability measure (r = −0.13, p = 0.02) and symptom dimensions of PANSS (r between −0.12 and −0.29, p < 0.05), except for positive symptoms (r = −0.10, p > 0.05). Years of education (β = 0.12, 95% CI (0.09, 0.14), p < 0.001), male sex (β = 0.22, 95% CI (0.05, 0.39)), age β = −0.02, 95% CI (−0.03, −0.01), and medication side effects (β = −0.03, 95% CI (−0.06, −0.01), p = 0.021) were significantly associated with the composite score of ECAS. The Item Response Theory analysis showed that the tool best functions among participants with moderate cognitive impairment (difficulty coefficient between −1.12 and 0.27). The Differential Item Functioning analyses showed that education had a positive contribution on Digit Symbol Substitution Test (MH OR = 2.64, 95% CI (1.34, 5.20)). The results showed that ECAS is valid in assessing cognition in PWS in low-resource settings.

Examining Differential Item Functioning from a Multidimensional IRT Perspective.

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Relating violations of measurement invariance to group differences in response times.

Measurement invariance is an assumption underlying the regression of a latent variable on a background variable. It requires the measurement model parameters of the latent variable to be equal across the levels of the background variable. Item-specific violations of this assumption are referred to as differential item functioning and are ideally substantively explainable to warrant theoretically valid and meaningful results. Past research has focused on developing statistical approaches to explain differential item functioning effects in terms of item- or person-specific covariates. In this study, we propose a modeling approach that can be used to test if differences in item response times can be used to statistically explain differential item functioning. To this end, we operationalize a latent response process factor and test if item-specific group differences on this factor can account for the observed differences in item scores. We investigate the properties of the model in a simulation study, and we apply the model to a real data set. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Evaluating the Validity and Measurement Invariance of the Patient Health Questionnaire-9 Among Canadian Nurses.

Background and Purpose: The patient health questionnaire 9 item (PHQ-9) is a widely used self-reported measure for screening depressive symptoms. This study aims to examine measurement invariance and psychometric properties of the PHQ-9 for screening depressive symptoms in nurses across different nursing roles, gender, and workplace sector. Methods: The study is a secondary analysis of pre-COVID and COVID-19 survey data from 4,176 nurses and 3,238 nurses in British Columbia. Data from the PHQ-9 tested the assumption of unidimensionality, reliability, and presence of differential item functioning (DIF). Results: The PHQ-9 showed excellent internal consistency (r = .9) and a unidimensional factor structure. PHQ-9 items were free of DIF across nursing roles, gender, and workplace sector. Conclusions: This study supported the valid use of the PHQ-9 as a screening tool for depressive symptoms among nurses.

Development and initial validation of the Multifaceted Instrument for Body Image Disturbance (MI-BoD).

Specific facets of body image disturbance have an important role in eating disorder (ED) onset and maintenance yet have been assessed with single items and/or questionnaires predominantly developed in female samples to capture desire for a thinner body. The aim of this multipart study was to develop the multifaceted instrument for body image disturbance (MI-BoD) that will assess body image disturbance across gender and body size. In Study 1, interviews were conducted with 31 ED lived experience experts (45% females) to develop an initial item pool which was then rated for relevance and clarity by 59 international field experts, shortened and refined by the research team, and then rerated by 20 field experts and 91 ED lived experience experts. Of the 477 items developed from interviews, 46 were retained for the initial validation study. In Study 2, the MI-BoD was administered to undergraduate students (N = 937; 84% females), community adolescents (N = 208; 58% females), and individuals with self-reported ED diagnosis (N = 410; 77% females) to assess its preliminary psychometric properties. Exploratory factor analysis revealed six underlying factors, namely, Dissatisfaction, Overvaluation, Preoccupation, Fear of Weight Gain, Body Checking, and Body Exposure. No differential item functioning was detected for most MI-BoD items across gender, weight status, and ED status (symptomatic vs. asymptomatic). Overall, the MI-BoD showed good internal consistency, convergent and divergent validity, concurrent validity, and test-retest reliability. In conclusion, the MI-BoD is a promising tool for assessment of important facets of body image disturbance across gender, body size, and ED symptomatology. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Age-Neutral Measurement Of Personality Functioning and Maladaptive Personality Traits.

As previous studies have shown that personality disorder (PD) assessment in older adults is often hampered because assessment tools are tailored toward younger adults, establishing the age-neutrality of novel tools is crucial. This study primarily aimed to evaluate the age-neutrality of the Level of Personality Functioning Brief Form (LPFS-BF 2.0) and the Personality Inventory for DSM-5 Modified + (PID-5-BF+M), using a sample of 254 community-dwelling adults. The analysis of Differential Item Functioning (DIF) demonstrated the age-neutrality of both instruments, with only 8.3% of LPFS-BF 2.0 items and 5.6% of PID-5-BF+M items exhibiting DIF. Differential Test Functioning (DTF) analyses revealed large DTF for the LPFS-BF 2.0 total score, indicating that age-specific norms might be necessary for this score. In summary, this study supports the use of these instruments in both older and younger adults, enhancing the assessment of PDs across the life span.

Development of a rapid tool for screening financial toxicity based on the comprehensive score for financial toxicity

ObjectiveThis study aimed to use item response theory (IRT) to develop and validate a short version of the Chinese COmprehensive Score for financial Toxicity (COST-S-C) in patients with cancer. MethodsA total of 500 patients with cancer participated in the survey. Confirmatory factor analysis (CFA) was used to assess the factor structure of the COST-S-C. A two-parameter IRT model was used to examine the item fit of the COST-S-C. Item discrimination, item characteristic curves (ICCs), item information curves (IICs), and item fit were estimated. Differential item functioning (DIF) was assessed for participants’ gender and age. Reliability, convergent and discriminant validity was examined as well as estimating the optimal cut-off points of the COST-S-C. ResultsThe CFA results supported the bi-factor structure of the COST-S-C. The IRT analysis findings revealed that several items showed problems with the ICCs, providing little information in terms of IICs as well as poor discrimination. Item 5 showed a negligible DIF problem with age. A valid 3-item COST-S-C scale and its optimal cut-off point was developed. ConclusionsThe COST-S-C is a valid and quick screening tool used to distinguish patients with cancer who report significant financial toxicity in various medical settings.

Psychometric validation of the Danish version of the Major Depression Inventory using data from the Lolland-Falster health study (LOFUS)

Purpose The Major Depression Inventory (MDI) is a widely used self-rating depression scale commonly in primary care in Denmark. It has not been subject to robust psychometric validation in a general population setting. The aim of this study was to evaluate the psychometric measurement properties of the MDI when applied in the general population. Methods We evaluated statistical psychometric validity using modern test theory (confirmatory factor analysis, item response theory models and Rasch measurement theory) testing local independence and differential item function across groups defined by gender, age, education, and chronic disease status. Separate analyses across different strata and across different statistical models were employed. Results Regarding structural validity we consistently identified local dependence for the item two pairs (MDI2,MDI3) and (MDI4,MDI5) across strata. This result was confirmed by bifactor CFA models and item screening. We further identified substantial differential item functioning with respect to age group and with respect to chronic disease. We identified quantified the magnitude of this lack of measurement invariance. Conclusion The MDI is psychometrically valid in homogenous sub populations, but the disclosed evidence of local dependence means that published estimates of its reliability cannot be trusted. The lack of measurement invariance means that the instrument cannot be used to compare individuals or groups unless they are similar in terms of age group and chronic disease status.

A short executive functioning questionnaire in the context of early childhood screening: psychometric properties

ABSTRACT Early childhood executive functioning (EF) predicts later adjustment and academic achievement. However, measuring EF consistently and efficiently across settings in early childhood can be challenging. Most researchers use task-based measures of EF, but these methods present practical challenges that impede implementation in some settings. The current study of 380 3–5-year-old children in the United States evaluated the psychometric properties of a new 14-item parent-reported measure of EF in a diverse urban school district. This questionnaire aimed to capture a normative range of EF skills in ecologically valid contexts. There was evidence for two specific subscales – one that measures children’s EF challenges and another that measures children’s EF skills. Results suggested that several items demonstrated differential item functioning by age and race. After adjusting for measurement differences across demographic groups and controlling for age at screening, the EF challenges subscale was more strongly related to task-based measures of EF than was the EF skills subscale. EF challenges predicted third-grade math achievement, controlling for demographic variables and a performance-based measure of children’s early cognitive and academic skills. Results suggest that this parent report of EF could be a useful and effective early childhood screening tool.

Investigating measurement invariance in PIRLS 2021 across English and isiZulu language groups

Background: PIRLS 2021 results revealed that South African Grade 4 learners performed significantly lower compared to other countries in reading comprehension and that they did not reach the standardised international mean score of 500. It was also evident from the results that English learners performed relatively higher than isiZulu learners.Objective: This study was initiated to investigate whether possible measurement invariance can help explain the difference in the scores of English and isiZulu learners. Measurement invariance is concerned with the metric and functional equivalence of the items in the two languages examined.Method: A quantitative approach was utilised to reanalyse the PIRLS 2021 numerical data sets. Descriptive statistics were used to analyse the mean and raw scores. Rasch analysis was conducted to look for any Differential Item Function (DIF) in the items.Results: Statistics revealed that there is a significant difference in the mean scores of the two languages. Analysis of raw scores provided evidence that some of the items lacked metric equivalence. Rasch analysis displayed that some of the items lacked functional equivalence.Conclusions: The significant difference found in the scores implies that Grade 4 learners need more attention mastering the skill of reading literacy. The lack of metric and functional equivalence suggests that items in the passages need to be investigated and improved.Contributions: This study has contributed to the International Large-Scale Assessment literature specifically relating to the equivalence of ILSA across languages. A significant gap in the learners’ scores was identified and measurement invariance helped explain the gap in the learners’ reading scores.

Understanding the alternative Mantel-Haenszel statistic: Factors affecting its robustness to detect non-uniform DIF

Test-item bias has become an increasingly challenging issue in statistics and education. A popular method, the Mantel-Haenszel test, is used for detecting non-uniform differential item functioning (DIF) but requires constructing several performance tiers to maintain robustness. The alternative Mantel-Haenszel (AMH) test was developed within the last three decades as a proxy procedure requiring only two scoring tiers. However, there needs to be more information on how essential factors like comparison group sizes and question discrimination affect its ability to detect bias. This study investigates how item difficulty and discrimination and the examined ratio between the focal and reference groups impact the likelihood of the AMH test detecting DIF. A comprehensive simulation study was conducted where the test scores were generated under three conditions. There are three commonly used difficulty levels (easy, medium, and hard), two discrimination levels (referred to as 'low’ and 'high’), and three group comparison ratios (1:1, 2:1, and 5:1). The simulation study showed the AMH test’s detection rates comparable to those of other standardized procedures like the Breslow-Day (BD) and even better. The current study aims to investigate and determine the factors that affect AMH test detection behavior. The study concludes with an application involving collegiate-level test data comparing students across genders and majors.

Item Response Theory Validation of the Forgotten Joint Score for Persons Undergoing Total Knee Replacement.

The Forgotten Joint Score (FJS), a commonly used patient-reported outcome measure, was developed without fully confirming assumptions such as unidimensionality (all items reflect 1 underlying factor), appropriate weighting of each item in scoring, absence of differential item functioning (in which different groups, e.g., men and women, respond differently), local dependence (pairs of items are measuring only 1 underlying factor), and monotonicity (persons with higher function have a higher score). We applied item response theory (IRT) to perform validation of the FJS according to contemporary standards, and thus support its ongoing use. We aimed to confirm that the FJS reflects a single latent trait. In addition, we aimed to determine whether an IRT model could be fitted to the FJS. Participants undergoing primary total knee replacement provided responses to the FJS items preoperatively and at 6 and 12 months postoperatively. An exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and Mokken analysis were conducted. A graded response model (GRM) was fitted to the data. A total of 1,774 patient responses were analyzed. EFA indicated a 1-factor model (all 12 items reflecting 1 underlying trait). CFA demonstrated an excellent model fit. Items did not have equal weighting. The FJS demonstrated good monotonicity and no differential item functioning by sex, age, or body mass index. GRM parameters are reported in this paper. The FJS meets key validity assumptions, supporting its use in clinical practice and research. The IRT-adapted FJS has potential advantages over the traditional FJS: it provides continuous measurements with finer granularity between health states, includes individual measurement error, and can compute scores despite more missing data (with only 1 response required to estimate a score). It can be applied retrospectively to existing data sets or used to deliver individualized computerized adaptive tests. Prognostic Level II. See Instructions for Authors for a complete description of levels of evidence.

A Bayesian Moderated Nonlinear Factor Analysis Approach for DIF Detection under Violation of the Equal Variance Assumption

AbstractResearch has shown that multiple‐indicator multiple‐cause (MIMIC) models can result in inflated Type I error rates in detecting differential item functioning (DIF) when the assumption of equal latent variance is violated. This study explains how the violation of the equal variance assumption adversely impacts the detection of nonuniform DIF and how it can be addressed through moderated nonlinear factor analysis (MNLFA) model via Bayesian estimation approach to overcome limitations from the restrictive assumption. The Bayesian MNLFA approach suggested in this study better control Type I errors by freely estimating latent factor variances across different groups. Our experimentation with simulated data demonstrates that the BMNFA models outperform the existing MIMIC models, in terms of Type I error control as well as parameter recovery. The results suggest that the MNLFA models have the potential to be a superior choice to the existing MIMIC models, especially in situations where the assumption of equal latent variance assumption is not likely to hold.

Investigating Differential Item Functioning of an International Mathematics Competition Items across Gender Groups

Mathematical problem-solving competitions have existed for over a century. Scholars report the gender gap in these competitions. As a result, it is necessary to determine whether any score difference between gender groups is attributable to a genuine difference or is the result of the exam itself. Thus, the current study specifically examined bias in one of the well-known mathematics competitions: the Kangaroo Mathematics competition. Determining the fairness of Kangaroo mathematics competition items across gender groups is crucial for creating accurate comparisons and avoiding unintended construct irrelevant bias. To examine the bias, Differential Item Functioning (DIF) analyses were conducted using Logistic Regression, Mantel-Haenszel, and Item Response Theory Likelihood Ratio Test DIF detection methods. After a series of investigations, out of 336 items, it was concluded that these mathematics items were free of DIF and bias across the gender groups. Further implications were discussed in detail regarding the validity and bias.

Detecting Differential Item Functioning in Multidimensional Graded Response Models With Recursive Partitioning.

Differential item functioning (DIF) is a common challenge when examining latent traits in large scale surveys. In recent work, methods from the field of machine learning such as model-based recursive partitioning have been proposed to identify subgroups with DIF when little theoretical guidance and many potential subgroups are available. On this basis, we propose and compare recursive partitioning techniques for detecting DIF with a focus on measurement models with multiple latent variables and ordinal response data. We implement tree-based approaches for identifying subgroups that contribute to DIF in multidimensional latent variable modeling and propose a robust, yet scalable extension, inspired by random forests. The proposed techniques are applied and compared with simulations. We show that the proposed methods are able to efficiently detect DIF and allow to extract decision rules that lead to subgroups with well fitting models.

Rasch Calibration and Differential Item Functioning (DIF) Analysis of the School Well-Being Scale for Students

Rasch Calibration and Differential Item Functioning (DIF) Analysis of the School Well-Being Scale for Students